In the art of general classification of measurement data, raw data, e.g. point cloud data, imaging data (2D or 3D), multi-spectral data, radar data, lidar data, data by a total station, or data from a laser tracker, are transformed into higher-level geometrical and semantic representations, e.g. for surface reconstructions, geometry fitting, 3d-modelling in particular surface meshing, fitting of parametric 2D or 3D primitives, selective processing, data decimation, data visualization, benchmarking, and/or object detection or recognition.
Targets of interest within the measurement data are often classified into semantic and geometrical target classes, with possible hierarchical sub-classes, e.g.                semantic target classes                    terrain: man-made or natural            vegetation: high (trees) or low (bushes)            hardscape: buildings, cars, and remaining objects, e.g. wherein buildings can consist of wall, roof, balcony, window, door, etc.            application-specific classes: workpieces, products, tools, etc.            classes for filtering and/or data reduction, e.g. classes for automatically filtering out unwanted points like scanning artefacts caused by moving objects or vegetation which may not be relevant for projects in construction,                        geometrical target classes                    contour lines (“breaklines”), i.e. linear features where the curvature radius of the surface is small, i.e. along which the orientation (“normal vector”) of the underlying surface exhibits an unusual discontinuity            planar surfaces            objects with a predefined geometry (e.g. long vertical cylinders resembling light posts).                        
For example, point cloud classification might be based on feature vectors consisting of point- or region-wise properties, e.g. such as                geometry in local neighborhood, e.g. based on the eigenvalues/-vectors of the covariance matrix and/or moments of structure tensors in the neighborhood indicating properties like planarity, curvature, linearity, orientation and/or roughness        height information such as vertical range, height to ground/ceiling, etc.        intensity and/or color, and variations thereof.        
For example, classification on image data might be based on feature vectors consisting of local, region-wise or global properties, e.g. such as                histogram of oriented gradients or dense SIFT features (scale-invariant feature transform)        Haar-like features        Bag-of-Words features        image features based on image oversegmentation.        
In particular, in a machine learning framework for deep learning, e.g. based on Convolutional Neural Networks, features and classification rules are extracted automatically from training data.
The assignment of such classes and subclasses within measurement data still requires time-consuming processes and the human eye and human judgement. In attempt to automatize such assignment processes by computer implemented solutions many computational challenges are faced.
Often the acquired data are unstructured and highly inhomogeneous with strong variations in the point density, e.g. in a 3D point cloud, caused by a quadratic decrease of the point density with distance to the point cloud capturing device, as well as a decrease of laser/light intensity with distance and points falling below a detection threshold or points that are captured with reduced accuracy (low S/N). In a scanning dataset, specific scan patterns may be used with different spacing of scan points in x, y, z direction, or data sets may be incomplete due to occlusions.
In case of image data, variability of the appearance of an object might be very high as well. For example, object appearance depends on viewing direction, distance to the object, image resolution, lighting conditions and image sensor properties. Furthermore, in contrast to 3D point clouds an absolute scale of an object on the image is not defined.
Therefore, multiscale classification features describing short-, mid-, and long-range semantic and geometric properties, e.g. further including orientation and height information, are required, and computational algorithms need to cope with inhomogeneous and incomplete datasets. Moreover, since 10's to 100's of millions of data points or image pixels need to be processed, efficient algorithms, e.g. implementing parallel computing, and large data storage are required.
There are many applications with different definitions of classes. For example, for one application the classification of a point cloud into three classes (e.g. “man-made”, “vegetation” and “terrain”) might be sufficient, whereas another application might require splitting up vegetation in separate classes for “trees”, “bushes”, etc.
Then again, for data reduction it might be sufficient to only have a few selective classes, e.g. in a simple case filtering a dataset based on only two classes, i.e. “data points to keep” and “data points to filter out”.
Moreover, an object might look different if data are recorded at different places and during different seasons of the year. For example, since buildings in Asia may look quite different from buildings in Europe, a classifier that is purely trained based on data captured in Europe might not work well in other regions of the world.
Therefore, great efforts relating to manual classification steps, e.g. manual data filtering and manual data assignment, are required. On the other hand, these ongoing efforts may be beneficially used as a departure point for an inventive classification and/or training workflow based on machine learning.
Application of machine learning algorithms allows an automation of different processes in classifying measurement data. Such a classification framework, based on a subclass of general machine learning (ML), provides a very efficient “learning approach” for pattern recognition as compared to rule-based programming. Machine learning algorithms can deal with tasks of large complexity, make use of implicit or explicit user feedback, thus becoming adaptive, and provide “per point” probabilities of the classification. This saves time, reduces processing costs and decreases amount of manual work.
In so-called “supervised ML” an algorithm implicitly learns which characterizing properties (i.e. a combination of features) define target properties of points (such as class membership, affiliation to a contour line, etc.) according to definitions made by the user when labelling training data.
On the other hand, in so-called “unsupervised ML” the algorithm finds hidden structure in unlabeled data, e.g. based on the data alone or with the aid of additional information such as for example a class specific (a-priori) model based on a specific classifier or a set of classifiers. This is also called “clustering” or “segmentation” and involves grouping points of the measurement data into categories based on some measure of inherent similarity or distance.
Probabilistic classification algorithms further use statistical inference to find the best class for a given instance. Instead of simply determining a “best” class for each instance, probabilistic algorithms provide a probability of the instance being a member of each of the possible classes, wherein normally the one class with the highest probability is selected. This has several advantages over non-probabilistic algorithms, i.e. associating a confidence value for weighting its choice, and consequently, providing an option to abstain a choice when its confidence value is too low.
However, usage of machine learning algorithms requires a lot of training data. In case of supervised machine learning also labeling information (i.e. assignment of the object classes to the data) is necessary. The data acquisition, preparation and labeling requires a lot of efforts and time.
Summarizing, it is difficult in practice to use one pre-trained classifier for large varieties of different applications, definitions of classes, object's appearance, etc.
Even with increasing computing power and data storage, the large complexity required for automatically assigning particular classes of interest within measurement data pushes traditional methods, such as rule-based computational methods, to their limits, and a high degree of human interaction is still required.